Locating wireless devices

ABSTRACT

Systems, methods, devices and non-transitory, computer-readable storage mediums are disclosed for location-tracking wireless devices. In an embodiment, a method performed by an electronic device comprises: playing, or initiating the playing of, a sound through a loudspeaker of an accessory device via a communication link. The sound is played at a specified frequency that utilizes a frequency response of the loudspeaker (or loudspeaker plus speaker enclosure). The sound is received through two or more microphones of the electronic device and filtered by one or more filters. The one or more filters are configured to pass the sound at or around the specified frequency and to reduce masking of the sound by ambient noise. The filtered sound is associated with direction data generated from sensor data provided by one or more inertial sensors of the electronic device. In another embodiment, the specified frequency is higher than the maximum human hearing range.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.17/010,473, entitled “Locating Wireless Devices,” filed Sep. 2, 2020,which is a continuation of U.S. patent application Ser. No. 15/721,666,entitled “Locating Wireless Devices,” filed Sep. 29, 2017, which claimsthe benefit of U.S. Provisional Patent Application No. 62/444,295,entitled “Locating Wireless Devices,” filed Jan. 9, 2017, the entirecontents of each of which are incorporated herein by reference.

TECHNICAL FIELD

This disclosure relates generally to locating wireless devices that havebeen lost or stolen.

BACKGROUND

Wireless devices, such as smart phones and tablet computers, ofteninclude a client application that allows a user to access anetwork-based service for remote location-tracking of the mobile deviceafter it has been lost or stolen. The user can invoke a “lost mode”remotely through a server computer using another computer or device toflag the device as lost or stolen. If the wireless device includespositioning technology, the device can be configured to report its lastlocation to the server computer, which is displayed by the service on amap presented to the user.

Often wireless devices are used with wireless accessory devices thatcannot determine their location and cannot communicate with a remotetracking services over a wide area network, such as the Internet. Theseaccessory devices can include, for example, wireless earbuds,headphones, headsets and other wearable devices (e.g., smartwatches,fitness bands, optical head-mounted displays) that communicate directlywith the wireless device using peer-to-peer communications. Sometimes auser misplaces a wireless accessory device and would like assistance inlocating the wireless accessory device. The currently available remotelocation-tracking services, however, are unable to track lost or stolenaccessory devices.

SUMMARY

Systems, methods, devices and non-transitory, computer-readable storagemediums are disclosed for locating wireless devices.

In an embodiment, a method performed by an electronic device comprises:playing, or initiating the playing of, a sound through a loudspeaker ofan accessory device via a communication link. The sound is played at aspecified frequency that utilizes a frequency response of theloudspeaker (or loudspeaker plus speaker enclosure). The sound isreceived through two or more microphones of the electronic device(configured for beamforming). A beamforming signal is generated fromoutputs of the two or more microphones, which is filtered by one or morefilters. The one or more filters are configured to pass the sound at oraround the specified frequency and to reduce masking of the sound byambient noise. The filtered sound is associated with orientation datagenerated from sensor data provided by one or more inertial sensors ofthe electronic device. In another embodiment, the specified frequency ishigher than the maximum human hearing range.

In an embodiment, a system comprises: one or more processors; memoryoperable to store instructions, which, when executed by the one or moreprocessors, causes the one or more processors to perform operationscomprising: playing, or initiating the playing of, a sound through aloudspeaker of an accessory device, the sound played at a specifiedfrequency that utilizes a frequency response of the loudspeaker;receiving through two or more microphones, the sound played by theloudspeaker; generating a beamforming signal from outputs of the two ormore microphones; filtering, by one or more filters, the beamformingsignal, the one or more filters configured to pass the sound at oraround the specified frequency and to reduce masking of the sound byambient noise; and associating the filtered sound with a compass headingfor the electronic device generated from sensor data provided by one ormore inertial sensors.

In an embodiment, a non-transitory, computer-readable storage mediumhaving instructions stored thereon causes one or more processors toperform operations comprising: playing, or initiating the playing of, asound through a loudspeaker of an accessory device, the sound played ata specified frequency that utilizes a frequency response of theloudspeaker; receiving through two or more microphones, the sound playedby the loudspeaker; generating a beamforming signal from outputs of thetwo or more microphones; filtering, by one or more filters, thebeamforming signal, the one or more filters configured to pass the soundat or around the specified frequency and to reduce masking of the soundby ambient noise; and associating, by the electronic device, thefiltered sound with a compass heading generated from sensor dataprovided by one or more inertial sensors.

In an embodiment, an electronic device comprises: two or moremicrophones; one or more inertial sensors; a wireless communicationsinterface; one or more processors; memory operable to storeinstructions, which, when executed by the one or more processors, causesthe one or more processors to perform operations comprising: playing, orinitiating the playing of, a sound through a loudspeaker of an accessorydevice, the accessory device communicatively coupled to the electronicdevice through the wireless communication interface, the sound played ata specified frequency that utilizes a frequency response of theloudspeaker; receiving, by the electronic device through the two or moremicrophones of the electronic device, the sound played by theloudspeaker; generating a beamforming signal from outputs of the two ormore microphones; filtering, by one or more filters, the beamformingsignal, the one or more filters configured to pass the sound at oraround the specified frequency and to reduce masking of the sound byambient noise; and associating the filtered sound with an orientation ofthe electronic device generated from sensor data provided by the one ormore inertial sensors.

In an embodiment, a method comprises: causing, by an electronic device,a graphical user interface (GUI) to be displayed on a display of theelectronic device, the GUI including a first graphical elementindicating a first sound pressure level associated with a firstorientation of the electronic device in a reference coordinate system,wherein at least one dimension of the first graphical element in the GUIis determined at least in part by the first sound pressure level; andcausing, by the electronic device, a second graphical element to bedisplayed in the GUI, the second graphical element indicating a secondsound pressure level that is different than the first sound pressurelevel, the second sound pressure level associated with a secondorientation of the electronic device in the reference coordinate system,wherein at least one dimension of the second graphical element in theGUI is determined at least in part by the second sound pressure level.

Particular implementations disclosed herein provide one or more of thefollowing advantages. Users are provided with assistance in locating alost or stolen wireless accessory device by playing, or initiate playingof, a sound through a loudspeaker of the wireless accessory device at aspecified frequency during a sound playing mode. The playing can beinitiated by another electronic device in the user's possession(hereinafter referred to as a “companion device”) that iscommunicatively coupled to the wireless accessory device directly orindirectly through a communication link. The specified frequency can bethe resonant frequency of the loudspeaker (or the loudspeaker and aspeaker enclosure) to enable the sound to be more easily heard by theuser and detected by the companion device in an environment with ambientnoise. One or more narrow band filters can be tuned to pass the resonantfrequency and to attenuate other frequencies.

In an embodiment, the user of a companion device is assisted in locatingthe accessory device by a sound locator graphical user interface (GUI)that is displayed on the companion device that indicates a direction ofthe sound source using sound pressure level (SPL) measurements anddirection data (e.g., heading data) provided by, or calculated from,inertial sensor data (e.g., gyroscope data, magnetometer data). In anembodiment, radio frequency (RF) data, such as received signal strengthindicator (RSSI) values provided by a wireless communications interfaceof the companion device are also used to update the sound locator GUI.In an embodiment, spatial signal processing (e.g., a beamforming) can beused to direct the gain of two or more microphones of the companiondevice in a preferred direction to improve detection of the sound by thecompanion device and to attenuate ambient noise from the environment.

In an embodiment, a microphone of the wireless accessory device can beturned on in a listening mode to allow the user to hear the ambientsounds of the environment where the accessory device is located andleverage their knowledge of the environment to determine where theaccessory device may be located. The listening mode can be timemultiplexed with the sound playing mode to allow alternating betweenlistening to the played sound on the lost device and listening to theambient sounds of the environment.

The details of the disclosed implementations are set forth in theaccompanying drawings and the description below. Other features, objectsand advantages are apparent from the description, drawings and claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a system for locating a wireless accessorydevice, according to an embodiment.

FIG. 2 illustrates a frequency response of an example loudspeaker of anexample wireless accessory device, according to an embodiment.

FIG. 3 illustrates human ear sensitivity as a function of frequency andintensity, according to an embodiment.

FIG. 4 is a block diagram of system for locating a wireless accessorydevice, according to an embodiment.

FIG. 5 illustrates an example use case where a user loses her wirelessearbud in her home and then uses her smartphone to locate the earbud,according to an embodiment.

FIG. 6 is a flow diagram of a first process for locating a wirelessaccessory device, according to an embodiment.

FIG. 7 is a flow diagram of a second process for locating a wirelessaccessory device, according to an embodiment.

FIG. 8 is a flow diagram of a third process for locating a wirelessaccessory device, according to an embodiment.

FIG. 9 is a flow diagram of a fourth process for locating a wirelessaccessory device, according to an embodiment.

FIG. 10A illustrates a first GUI of a companion device for locating awireless accessory device on a map, according to an embodiment.

FIG. 10B illustrates a second GUI of a companion device for locating awireless accessory device and initiating a sound playing mode, accordingto an embodiment.

FIG. 10C illustrates a third GUI of a companion device for locating awireless accessory device, including a feature for independently mutingleft and right loudspeakers of the wireless accessory device, accordingto an embodiment.

FIG. 10D illustrates a fourth GUI of a companion device for locating awireless accessory device, including a sound locator GUI, according toan embodiment.

FIG. 10E illustrates a fifth GUI of a companion device for locating awireless accessory device, including a sound GUI with the beamformer ofthe companion device oriented in the opposite direction of the wirelessaccessory device, according to an embodiment.

FIG. 10F illustrates a sixth GUI of a companion device for locating awireless accessory device, including a sound GUI with the beamformer ofthe companion device oriented in the direction of the wireless accessorydevice, according to an embodiment.

FIG. 11 is a server computer architecture, according to an embodiment.

FIG. 12 is a companion device architecture, according to an embodiment.

FIG. 13 is a wireless accessory device architecture, according to anembodiment.

The same reference symbol used in various drawings indicates likeelements.

DETAILED DESCRIPTION

Example Systems

FIG. 1 is a block diagram of a system for locating a wireless accessorydevice, according to an embodiment. System 100 includes one or moreserver computers 102, requesting device 104, network 106, companiondevice 108 and wireless accessory device 110. Server computer(s) 102 canbe configured to provide a network-based device locator service. In anembodiment, server computer(s) 102 implement a website that allows auser of a requesting device 104 to configure a lost mode on a wirelessdevice.

Requesting device 104 can be any electronic device or computer withnetwork connectivity, including but not limited to: desktop computers,laptop computers, smartphones, tablet computers, wearable devices andthe like. In an embodiment, requesting device 104 is also companiondevice 108 and can assist the user in finding their lost wirelessaccessory device, as described in further detail below.

Wireless accessory device 110 can be any device that communicates eitherdirectly or indirectly (e.g., through another device or computer) with acompanion device over a wireless network or peer-to-peer communicationlink. Some examples of wireless accessory devices include but are notlimited to wireless earbuds, headphones, headsets and other wearabledevices (e.g., smartwatches, fitness bands, optical head-mounteddisplays, game controllers, remote controls, smartphones). In anembodiment, wireless accessory device 110 can be “paired” with companiondevice 108 according to, for example, a wireless technology standard,such as Bluetooth.

Some examples of wireless networks include a wireless local area network(WLAN), such as the Internet, and a wireless personal area network(WPAN). Some examples of communication technologies include any suitableIEEE 802.xx technology, including but not limited to Wi-Fi andBluetooth.

Companion device 108 can be any electronic device capable ofcommunicating with a wireless accessory device (e.g., a device thatincludes a wireless transceiver chip), and that includes at least onemicrophone, or other sound transducer, and a display or other outputdevice (e.g., an audio output device) for assisting a user in locating awireless accessory device. Some examples of a companion device includebut are not limited to: a smartphone, a tablet computer, a notebookcomputer, a wearable computer (e.g., a smartwatch) and the like.

In operation, a user logs into a device locator service on servercomputer(s) 102 using requesting device 104. The user is presented witha GUI which allows the user to locate a lost or stolen accessory device110 on a map. For example, the GUI could present the map with a markershowing the location of the wireless accessory device. The devicelocator service establishes communication through network 106 withcompanion device 108. Companion device 108 includes one or morepositioning technologies, such as a global navigation satellite system(GNSS) receiver chip or a terrestrial positioning system using RFsignals from access points (APs) of a wireless network or cell towertransmitters of a cellular telephone network. An example GNSS system isthe Global Positioning System (GPS). In an embodiment, companion device108 periodically stores its own location. When the device locatorservice requests the location of wireless accessory device 110,companion device 108 determines if there is an operational communicationlink with wireless accessory device 110. For example, if thecommunication link is a Bluetooth communication link, companion device108 determines if companion device 108 is currently “paired” (e.g., inwireless communication) with wireless accessory device 110. If so, thencompanion device 108 returns its current location as the location ofaccessory device 110. If, however, there is no operational communicationlink with accessory device 110, companion device 108 returns thelocation that corresponds to a time when connectivity with accessorydevice 110 was lost or the location that corresponds to a time whenaccessory device 110 was last connected or used. For example, each timeconnectively is lost with accessory device 110 (a disconnect event),companion device 108 stores its current location in memory. In anembodiment, a history of locations and times of disconnect events can bestored by companion device 108. In other embodiments, only a currentlocation of companion device 108 at the time of the last disconnectevent is stored.

In an embodiment, when user initiates on a companion device orrequesting device a sound to be played on an accessory device, some orall of the companion devices that were previously connected to theaccessory device (e.g., through a Bluetooth or other connection) aresent a notification from the server computer that includes a pendingplay sound command. For example, a push notification or otherInternet-based communication can be sent to the companion devices, wherethe command is initiated by a central server computer.

When the companion devices receive the pending play sound command, thecompanion devices each scan (e.g., using Bluetooth) to find theaccessory device over a period of time. If the accessory device is foundby a companion device, the companion device will contact the servercomputer to determine if the sound still needs to be played on theaccessory device. If the sound still needs to be played, the companiondevice will establish a connection with the accessory device (e.g.,using Bluetooth) and play the sound on the accessory device or invokethe accessory device to play the sound. When the accessory device startsplaying the sound, all of the companion devices that were once connectedto the accessory device will receive notifications that will cause amessage or other indicator to be displayed on each companion device(e.g., on the lock screen), informing the user that the sound is playingon the accessory device. The user can then stop the sound using agraphical element provided by or presented in the notification, orinvoke a device locator client application to view the location of theaccessory device on a map and/or mute or stop the sound from playing onthe accessory device, as described in reference to FIGS. 10B and 10C.

In another embodiment, one or more of received signal strength values(e.g., RSSI values) (or average RSSI values), estimated ranges or rangeclasses (e.g., Nearby, Far) that are measured or calculated on thecompanion devices are sent to the server computer periodically or inresponse to a trigger event. The server computer can then compare theseproximity values to determine which companion device is closest to theaccessory device. The server computer can then send the play soundcommand only to that companion device or a subset of companion devicesthat are within a specified proximity of the accessory device.

Although system 100 can show a user the location of wireless accessorydevice 110 on a map, it is also desirable to locally assist the user whois operating companion device 108 to find wireless accessory device 110.This can be accomplished using a sound locator application running oncompanion device 108 that can play a sound, or initiate the playing of asound, on a loudspeaker of wireless accessory device 110, and thendetermine the location of wireless accessory device 110 in the localenvironment based on the sound received through one or more microphonesof companion device 108, direction data (e.g., compass heading data) andoptionally RF data, which is described in further detail below inreference to FIGS. 2-10.

FIG. 2 illustrates a frequency response of an example loudspeaker of anexample accessory device, according to an embodiment. The shaded regionrepresents a population of frequency response curves. As shown in FIG.2, this example loudspeaker has a first peak 200 in the frequencyresponse at about 2.6 kHz. As shown in FIG. 3, 2.6 kHz is close to themaximum sensitivity range of the human ear, which is about 3-4 kHz.Human hearing does not have a flat spectral sensitivity or frequencyresponse relative to frequency versus amplitude. For example, humans donot perceive low and high frequency sounds as well as sounds between 3-4kHz, as shown by the equal-loudness contours in FIG. 3.

To ensure efficient power transfer to the loudspeaker in the accessorydevice, the companion device can stream an audio signal at about 2.6 kHzover the direct communication link (e.g., over a Bluetooth link). Awireless communications interface (e.g., a wireless transceiver chip) inthe accessory device can receive and decode (if encoded) the audiosignal and an audio processing unit in the accessory device can play thedecoded audio signal through a loudspeaker. The sound can be a tone or apattern of tones at the desired frequency. By playing a tone at or closeto the resonant frequency of the loudspeaker, maximum loudness can beachieved to assist the user or the companion device in detecting thesound in an environment with ambient noise, such as a bar or restaurant.Additionally, matching the sound frequency with the frequency responseof the loudspeaker is efficient in that less power will be needed toplay the sound at maximum loudness. Reducing power loss allows the soundto be played at a sustained maximum loudness over a longer period oftime than if played at another frequency, thus providing the user moretime to locate the accessory device. This is beneficial in that abattery-powered accessory device may be low in power at the time ofloss.

In an alternative embodiment, the frequency of the sound can be sethigher than the maximum human hearing range, for example, at about 20.5kHz. Playing the sound at an ultrasonic frequency eliminates thepossibility of annoying the user if the accessory device is in theuser's ear when the sound is played at maximum volume.

FIG. 4 is a block diagram of system for locating a wireless accessorydevice, according to an embodiment. System 400 can be implemented on acompanion device in software, firmware, hardware or any combinationthereof. For example, system 400 can be implemented using the examplecompanion device architecture, described in reference to FIG. 12. System400 can include wireless transceiver 402 and communication subsystem 404(collectively, a wireless communication interface), audio subsystem 406(including the beamformer), filter module 408, sound locator module 410,sensor subsystem 412, database 414 and microphone(s) 416. System 100 isan example system and other embodiments can include more or fewersubsystems. Although the subsystems in FIG. 4 are shown as separateblocks, in some embodiments, two or more subsystems can be combined intoa single subsystem and any individual subsystem can be divided intoadditional subsystems depending on the particular application orcompanion device architecture.

Audio subsystem 406 can be implemented by an audio processing unit toselect a suitable sound file from database 414 for playback on anaccessory device. In an embodiment, the sound file can be predeterminedand stored on the companion device or accessory device. The sound filecan be a generic sound file associated with a particular loudspeakermodel or can be a customized audio file for a specific loudspeaker. Inan embodiment, when a companion device “pairs” or otherwise is incommunication with an accessory device, the accessory device can providea device profile that includes information about the accessory device,such as the type of loudspeaker that is installed. Audio subsystem 406can then use this loudspeaker information to select a sound file thathas a tone or tone pattern to be played at or near the resonantfrequency of the loudspeaker, or the loudspeaker and its enclosure whichmay also influence the resonant frequency. In an embodiment, database414 can store a number of generic sound files for different loudspeakermodels. For example, a generic sound file can include a tone or patternof tones designed for an average resonant frequency for the loudspeakermodel or accessory device (loudspeaker plus enclosure) as determined bytesting over a number of devices of a particular model. In anotherembodiment, communication subsystem 404 can obtain a suitable audio filefrom a server computer (e.g., server computer(s) 102) through wirelesstransceiver (TX) 402. This latter example embodiment would allow eachindividual loudspeaker to have a customized sound file that can bedetermined during factory testing and stored in a network database,which can be accessed by the companion device.

After a suitable sound file is obtained, the sound file is sent tocommunication subsystem 404, which formats and streams the sound file tothe accessory device, where the sound file is received and played by aprocessor of the accessory device through a loudspeaker of the accessorydevice. In another embodiment, the sound file can be stored in memory ofthe accessory device or generated by the accessory device. In such anembodiment, audio subsystem 406 can generate a signal or command that issent to the accessory device through a direct or indirect communicationlink. The signal or command is configured to cause the processor of theaccessory device to play the sound file through the loudspeaker. Anexample wireless accessory device architecture is described in referenceto FIG. 13.

The sound that is played by the wireless accessory device is received byone or more microphones 416 coupled to audio subsystem 406. In anembodiment, audio subsystem 406 can configure the two or moremicrophones for beamforming (BF), or use an embedded directionalmicrophone (e.g., a cardioid microphone). Beamforming applies spatialfiltering as a means to receive sound preferentially in a direction ofinterest. Beamforming is used to increase the signal-to-noise (SNR)ratio of the detected sound signal by attenuating ambient noise outsidethe direction of interest. In an embodiment, two microphones 416 locatedon a front or top edge of the companion device are configured to have acardioid beam pattern to implement beamforming. For example, the mainlobe of the beam pattern is projected in front of the user as she walkstowards the accessory device.

In an embodiment, audio subsystem 406 monitors ambient sound from theenvironment for specified period of time (e.g., 5 seconds) to calculatea benchmark average sound pressure level (SPLR). SPLR is defined as theRMS value of the instantaneous sound pressures measured over a specifiedperiod of time in decibels (dB). The ambient sound can be monitored withthe two beamforming microphones or other microphones located elsewhereon the companion device. After calibration is complete, the sound isoutput to filter module 408 which filters the sound to reduce themasking effect of the ambient noise and increase the detectability ofthe sound. For example, a band-pass filter (BPF) centered around theresonant frequency of the loudspeaker (e.g., a passband centered around2.60 kHz) can be applied to the sound signal. The frequency of 2.6 kHzis only an example frequency and each loudspeaker and accessory deviceor pair of wireless accessory devices (e.g., a pair of earbuds) willhave their own frequency response which could have a resonant frequencythat is higher or lower than 2.6 kHz. In general, the frequency of thesound should be at or near the resonant frequency of the loudspeaker tomaximize loudness and efficient use of power to provide the maximumloudness. In an embodiment, two consecutive narrow band-pass filters areapplied to the received sound signal. The first filter allowsfrequencies between about 2.5 kHz and about 2.7 kHz to pass. The secondfilter attenuates ambient noise even more to obtain a clear 2.6 kHzsignal. Parameters for the filter characteristics can be stored indatabase 414 as part of an accessory device profile or obtained from aserver computer.

Next, a normalized SPL_(N) is calculated based on the benchmark averageSPLR level using, for example, Equation [1]:

$\begin{matrix}{{{SPL}_{N} = {20{\log_{10}\left( \frac{P}{P_{0}} \right)}{dB}}},} & \lbrack 1\rbrack\end{matrix}$

where P is the root-mean-square (RMS) value corresponding to the currentsound pressure and P₀ is the RMS of the benchmark average SPLR.

The normalized SPL_(N) value is used to display sound strengthindicators in a sound locator GUI, as described below and in referenceto FIG. 10D. The normalized SPL_(N) value is provided to sound locatormodule 410, which generates and updates the sound locator GUI.

The sound locator module 410 associates the normalized SPL_(N) valuewith a direction (e.g., a heading in degrees) obtained from sensorsubsystem 412. Sensor subsystem 412 is coupled to one or more inertialsensors, including but not limited to gyroscope 418 a, accelerometers418 b and magnetometer 418 c. Sensor subsystem 412 obtains (e.g.,samples) raw sensor data from the one or more inertial sensors,optionally processes the sensor data (e.g., low pass filters andaverages the sensor data) and calculates direction data from the rawsensor data. For example, sensor subsystem 412 can calculate a compassheading for the companion device from the raw or processed sensor data.As the user moves around the environment (or moves their arm) and walkstowards the accessory device (the sound source), the sound locator GUIis updated according to new sensor data and new SPL_(N) values. Thesound locator GUI can be updated at a specified refresh rate (e.g., 60Hz). As described in reference to FIG. 10D, the sound locator GUI can beimplemented as virtual compass with a rotating plate and SPL_(N) barsfor assisting the user to locate the sound source (i.e., the wirelessaccessory device).

In an embodiment, in addition to normalized SPL_(N) values thecommunication subsystem 404 calculates RSSI values from received RFsignals transmitted by the accessory device (e.g., Bluetooth signals).The RSSI values provide a sense of proximity of the companion device tothe accessory device. The proximity can be displayed on the companiondevice according to range classes. In an embodiment, the range classesinclude but are not limited to: “Nearby,” “In the Room,” “In the house”and “Far.” Other range classes are also possible. For example, rangeclasses that refer to distance can be used (e.g., 5 m, 10 m, 15 m and 20m). In an embodiment, the range classes can be calculated from a radiowave distance formula or statistically using, for example, a probabilitydensity function (PDF) and/or cumulative distribution function (CDF) ofRSSI values. For example, windowed signal measurements obtained from RFsignals transmitted by the accessory device can be classified into rangeclasses that are defined by threshold values obtained from a RF signalpropagation model. A range class observation is then obtained byselecting a range class among a plurality of range classes based on apercentage of a total number of windowed signal measurements that areassociated with the range class. The range class observation is thenprovided as input to a state estimator that estimates a range class thataccounts for process and/or measurement noise.

Using the range classes (rather than absolute range determination) helpsmitigate problems associated with Bluetooth proximity detection inmultipath environments. Presenting the range classes to the user informsthe user that they are getting closer or further from the sound source.Range classes can be displayed or can be indicated by an audio sound orhaptic feedback as a function of range class. For example, a soundpattern can be played on a loudspeaker of the companion device withincreasing volume and/or frequency as a function of range class or avibration pattern can be induced by a haptic engine of the companiondevice with increasing frequency as a function of range class.Additionally, the RSSI values can be provided to sound locator module410 for use in updating the GUI to influence how the normalized SPL_(N)values are displayed, as described further in reference to FIG. 10D.

FIG. 5 illustrates an example use case where a user (Emily) loses herearbud in her home and then uses her smartphone to locate the lostearbud, according to an embodiment. FIG. 5 shows a floorplan 500 of ahouse. Emily's lost earbud 110 is in Room 2. There is also ambient noisein Room 2 due to television 501. Emily is in another room and iscarrying her smartphone 108 in her hand while viewing a sound locatorGUI to find her lost earbud 110, as previously described. Dashed line502 indicates a wireless transmission from smartphone 108 to earbud 110and dashed line 503 indicates a wireless transmission from earbud 110 tosmartphone 108. Example process steps that can be implemented bysmartphone 108 to assist Emily 501 in finding earbud 110 will now bedescribed in reference to FIGS. 6-9.

Example Processes

FIG. 6 is a flow diagram of a first process 600 for locating a wirelessaccessory device, according to an embodiment. Process 600 can beimplemented using, for example, companion device architecture 1200described in reference to FIG. 12.

Process 600 can begin by determining a sound file for Emily's earbud anda maximum loudness allowed by local regulations (602). For example, thecompanion device can refer to a profile associated with the earbud todetermine a sound file. A generic or custom sound file can be selectedto maximize loudness by playing a tone or pattern of tones at or nearthe resonant frequency of the loudspeaker (or the loudspeaker plus thespeaker enclosure).

Process 600 can continue by optionally playing a pre-recorded message onthe earbud (604). For example, the message can broadcast warning that aloud sound is about to be played through the loudspeaker of the earbud.In an embodiment, the message can be spoken in a language that is usedby Emily's smartphone. For example, a language profile stored on thesmartphone can be used to determine the language for the message. Themessage can be streamed by the smartphone to the earbud over a direct orindirect communication link or the message can be stored in memory onthe earbud and invoked by a signal or command from the smartphone or aserver computer.

Process 600 can continue by optionally determining an in-ear detector(IED) status and display the status on a display of the smartphone(606). For example, at least one of an optical sensor, proximity sensoror biometric sensor in the earbud can detect when the earbud is placedin or proximate to a human ear. The IED status can be a messagepresented on the smartphone display so that Emily does not play a soundthat could affect the hearing of a human ear. In an embodiment, if theIED status indicates an “in-ear” status, the smartphone can lock out theplay sound feature automatically until the IED status changes to “notin-ear.” The IED status can be encoded by one or more bits that are sentfrom the earbud to the smartphone via a direct or indirect communicationlink (e.g., included in a payload of a Bluetooth packet).

Process 600 can continue by playing a sound, or causing a sound to beplayed, through the loudspeaker of the earbud at increasingly higherlevels of loudness up to a maximum loudness level, but not higher than amaximum loudness level allowed by government regulations (608). Forexample, the sound can include one or more tones or a pattern of tonesat frequency that is at or near the resonant frequency of theloudspeaker. Alternatively, the sound can be played at a frequency thatis above the range of human hearing (e.g., greater than 20.5 kHz) toensure that the sound cannot affect the hearing of a human ear. Thesound can be played through the loud speaker at low volume and thenslowly ramp up in volume until the maximum loudness level of the deviceis reached or the maximum loudness level allowed by governmentregulation is reached, whichever level is lower. The sound can bestreamed by the smartphone to the earbud, or the smartphone (or a servercomputer) can send a signal or command to the earbud to play the sound.In the latter case, the sound file can be stored in memory on theearbud, as described in reference to FIG. 13. For process 600, Emily canuse her hearing to locate the earbud rather than using a sound locatorGUI.

An advantage of process 600 is that the frequency at which the sound isplayed can be selected to be either close to the maximum sensitivity ofthe human ear (e.g., 3-4 kHz) and peak at or near the resonant frequencyof the loudspeaker to efficiently provide maximize loudness, or thefrequency can be selected to be above the maximum range of human hearingto protect the hearing.

FIG. 7 is a flow diagram of a second process 700 for locating a wirelessaccessory device, according to an embodiment. Process 700 can beimplemented using, for example, the companion device architecturedescribed in reference to FIG. 12.

Process 700 can begin by determining a sound file for Emily's earbud anda maximum loudness allowed by regulations (702), optionally playing apre-recorded message through the loudspeaker of the earbud (704),optionally determining an IED status and displaying the status on thesmartphone (706) and playing a sound, or causing a sound to be played,through the loudspeaker of the earbud at increasingly higher levels ofloudness up to a maximum loudness level, but not higher than a maximumloudness level allowed by the government regulations (708). Accordingly,steps 702 through 708 of process 700 are the same as steps 602-608 ofprocess 600, described in reference to FIG. 6, and therefore need not bedescribed again.

Process 700 can continue by configuring two or more microphones of thesmartphone for beamforming (710). Beamforming can be used to increasethe SNR ratio of the detected sound signal by attenuating ambient noiseoutside the direction of interest. In an embodiment, two microphoneslocated on a front or top edge of Emily's smartphone are configured tohave a cardioid beam pattern such that the main lobe of the beam patternis projected in front of Emily as she walks towards the sound source.Other directional beamforming patterns can also be employed.

Process 700 can continue by filtering the sound received from theenvironment to reduce the masking effect of ambient noise and toincrease the detectability of the sound (712). In an embodiment, twoconsecutive narrow band-pass filters are applied to the sound to removeambient noise to obtain a clear sound signal.

Process 700 can continue by obtaining normalized sound pressure valuesfor a plurality of directions or headings (714). For example, anormalized SPL_(N) value can be calculated for a given orientation(e.g., a given direction or compass heading) and stored on thesmartphone using, for example, an array of 360 SPL_(N) values that canbe indexed using a heading, hash or other index. For example, there canbe one normalized SPL_(N) value for each degree in 360 degrees, asdescribed in reference to FIG. 10D. For example, each time a new compassheading is detected the compass heading can be rounded to the nearestdegree and used to index a corresponding position in the array. Forexample, if a compass heading of 35.6 degrees is detected, the compassheading would be rounded to 36 and used to access position 36 in thearray. If the position is empty, then the normalized SPL_(N) for thecompass heading is stored at position 36 of the array. If the positionis occupied with an old normalized SPL_(N) value, then the oldnormalized SPL_(N) value is replaced with the new normalized SPL_(N)value at position 36 of the array.

Process 700 can continue by updating a sound locator GUI using thenormalized SPL_(N) values (716). For example, the height of SPL_(N) barscan be determined by the new SPL_(N) values, as described in referenceto FIG. 10D.

Process 700 provides the same advantages as process 600, but alsoprovides an additional advantage of using beamforming and band-passfiltering to increase signal detectability and therefore increase thelikelihood of finding the lost earbud. Process 700 also has the addedadvantage of helping users with hearing impairment locate their lostaccessory devices, since process 700 does not rely on the user's hearingto locate the accessory device. For example, if Emily is hearingimpaired she could simply look at the sound locator GUI to determine adirection to walk in to find her lost earbud.

FIG. 8 is a flow diagram of a third process for locating a wirelessaccessory device, according to an embodiment. Process 800 can beimplemented using, for example, the companion device architecturedescribed in reference to FIG. 12.

Process 800 can begin by determining a sound file for Emily's earbud anda maximum loudness allowed by regulations (802), optionally playing apre-recorded message through the loudspeaker of the earbud (804),optionally determining an IED status and displaying the status on thesmartphone (806) and playing a sound, or causing a sound to be played,through the loudspeaker of the earbud at increasingly higher levels ofloudness up to a maximum loudness level, but not higher than a maximumloudness level allowed by the government regulations, configuringmicrophones of the smartphone for beamforming (810) and filtering thesound received from the environment to reduce the masking effect ofambient noise and to increase the detectability of the sound (812).Accordingly, steps 802 through 812 of process 800 are the same as steps702-712 of process 700, described in reference to FIG. 7, and thereforeneed not be described again.

Process 800 can continue by obtaining SPL_(N) values for a plurality ofdirections or headings and RF data (814). In an embodiment, range classestimates can be generated from RSSI values obtained from RF signals(e.g., Bluetooth signals) transmitted by the earbud. The range class canbe displayed on the sound locator GUI on Emily's smartphone, asdescribed in reference to FIG. 10D. In this example, the range class maybe “In the House” since Emily is separated from the earbud by Room 3.

Process 800 can continue by updating the sound locator GUI using thenormalized SPL_(N) values and the RF data (816). For example, the RSSIvalues can be used to scale SPL_(N) bar heights, as described byEquation [2].

Accordingly, process 800 provides the same advantages as processes 600and 700, but also provides an additional advantage of using RF data toprovide a range class estimation and to update the sound locator GUI.

FIG. 9 is a flow diagram of a fourth process for locating a wirelessaccessory device, according to an embodiment. Process 900 can beimplemented using, for example, the companion device architecturedescribed in reference to FIG. 12.

Process 900 can begin by pairing or otherwise establishing a direct orindirect communication link between Emily's smartphone and the earbud(902). Process 900 can continue by sending a signal or command to theearbud to open a microphone of the earbud (904), and then transmitwirelessly ambient sound captured by the microphone to the smartphonewhere it can be played through a loudspeaker or a headphone jack of thesmartphone (906). Using the present example, after pairing thesmartphone and earbud, Emily can use her smartphone to listen to theambient sounds/noises captured by the earbud microphone and transmittedwirelessly to the smartphone. When Emily hears the ambient sound/noise,she may receive a clue on which room the earbud is located based on herknowledge of her house floorplan 500. For example, Emily would hear thesound of television 501 and determine that the earbud was in Room 2.Emily could then walk into Room 2 and invoke one of processes 600, 700or 800 to locate the earbud in Room 2. To increase the likelihood offinding the lost earbud, process 900 can be combined with processes 700or 800, to perform bi-directional transmissions between the smartphoneand the earbud based on time multiplexing in the same communicationsession or communication channel. For example, in a sound playing mode,the sound is played by the earbud for a first period of time (e.g., 2-5seconds). The smartphone is then placed in a listening mode for a secondperiod of time (e.g., 2-5 seconds) where the ambient sound is capturedand transmitted to the smartphone. To protect Emily's privacy or otherfamily members while operating in listening mode, the smartphone can beconfigured to not persistently store or process the ambient soundcaptured by the earbud. In another embodiment, an audio message can beplayed on the earbud informing any individuals in earshot that there isa search for the location of the earbud underway. For example, the audiomessage could be: “Emily is searching for her lost earbud.” The audiomessage could be time multiplexed with the listening mode.

Example Sound Locator GUIs

FIG. 10A illustrates a first GUI of companion device 1000 (a smartphonein this example) for locating a wireless device on a map, according toan embodiment. The first GUI includes map 1001 with marker 1002designating a current or last known location of an accessory device. Inthe example shown, the last known location of Emily's lost earbud isindicated by marker 1002. Marker 1002 can be an icon, image, graphic orany other user interface element. Selectable cell 1003 can present adescription/name of the lost device and an estimated distance betweenthe accessory device and the current location of companion device 1000.

FIG. 10B illustrates a second GUI of companion device 1000 for locatinga wireless accessory device and initiating a sound playing mode,according to an embodiment. In response to the user selecting cell 1003(Emily's Earbud), the second GUI is presented which includes a zoomedview of map 1001 with marker 1002 centered in map 1001. Button 1004 canbe selected by the user to play a sound on the earbud, as described inreference to FIGS. 1-9. In this example, the Emily presses button 1104.

FIG. 10C illustrates a third GUI of companion device 1000 for locating awireless accessory device including a feature for independently mutingleft and right loudspeakers of the accessory device, according to anembodiment. Responsive to selection of button 1004, an image 1005 of theleft and right earbuds are displayed. Buttons 1006 a, 1006 b are alsodisplayed to allow the user to mute independently the left and rightearbuds. This feature comes in handy in the instance case where only oneearbud is lost or stolen. Button 1007 allows Emily to stop playing thesound.

FIG. 10D illustrates a sound locator GUI of companion device 1000 forlocating a wireless accessory device, according to an embodiment. Thesound locator GUI includes a graphical element 1008 comprising arotatable portion (e.g., a two-dimensional (2D) rotatable circularplate) with direction indicators 1009 a, 1009 b and SPL_(N) bars 1010.The direction indicators 1009 a (an arrow) and 1009 b (a beam or cone)indicate the current pointing direction (+X axis in this example) ofcompanion device 1000. A proximity indicator 1111 (e.g., “Nearby” inthis example) is presented in the center of the rotatable graphicalelement to present to the user an estimate of their proximity to theaccessory device. SPL_(N) bars 1010 extend radially along the perimeterof the rotatable portion. The height of each SPL_(N) bar is scaled toindicate the SPL value in a corresponding orientation of companiondevice 1000 in a reference coordinate system (e.g., a local level orgeodetic coordinate frame). An example reference coordinate system isthe East, North, Up (ENU) Cartesian coordinate system or Earth-Centered,Earth-Fixed (ECEF) Cartesian coordinate system. In an embodiment, theorientation can be represented by a direction indicating graphicalelement (e.g., a compass heading arrow) in the range of 0° to 360°.

In an embodiment, as Emily moves around and walks towards the ear bud(the sound source), the sound locator GUI is updated with new SPL_(N)values and for the new headings. Emily need not rotate a full range of0° to 360° to begin the sound location process. If Emily knows thegeneral location of the sound, Emily can scan the local environmentusing a left to right motion of her arm to generate or update SPL_(N)bars for various compass headings. The higher the bars the higher thenormalized SPL_(N) for that heading. In the example shown, the SPL_(N)bars at about 45 degrees from Emily's current heading are higher thanother SPL_(N) bars indicating to Emily that she should walk in thedirection indicated by the highest SPL_(N) bars. By contrast, there areno bars shown on the opposite side of the virtual compass. Thus, thebars location on the virtual compass and the height of the SPL_(N) barsprovide visual feedback to Emily on the direction of the sound source.The use of visual feedback is helpful to users with hearing impairmentor in environments with high ambient noise. In an embodiment, an audiosubsystem of the companion device can provide audio feedback thatindicates a distance or proximity to the sound source in addition to orin place of the visual feedback. In an embodiment, a haptic engine inthe companion device can provide haptic feedback (e.g., a vibration orvibration pattern) that indicates proximity or distance to the soundsource in addition to or in place of the visual feedback and/or audiofeedback. For example, as Emily gets closer to the sound source, afrequency of a vibration pattern can increase or the cadence of thepattern can change. In an embodiment, the rotatable portion of thegraphical element can be a three-dimensional (3D) object projected inthe GUI.

In an embodiment, when graphical element 1008 is refreshed, the heightof each SPL_(N) bar for a given orientation (each degree in the range of0° to 360°) is calculated using Equations [2] and [3]:

$\begin{matrix}{{barHeight} = {{MAX}\left( {\left( {{\log\;{f\left( {{volume}*{factor}} \right)}*\left( \frac{100}{proximity} \right)},{minBarHeight}} \right),} \right.}} & \lbrack 2\rbrack \\{{{barHeight} = {{MIN}\left( {{barHeight},{maxBarHeight}} \right)}},} & \lbrack 3\rbrack\end{matrix}$

where volume is the normalized SPL_(N) value for the given orientation(e.g., heading), proximity is an average of RSSI values (or a rangeestimated from an average of RSSI values using a radio wave distanceformula or propagation model to the accessory device computed from RSSIvalues), minBarHeight is a minimum bar height to ensure that the SPL_(N)bar is visible in the GUI, maxBarHeight is a maximum bar height toensure the SPL_(N) bar does not extend to far, the MAX( ) functioncompares its two input values and returns the larger of two inputvalues, the MIN( ) function compares its two input values and returnsthe smaller of the two input values, log f( ) is logarithmic fitfunction that fits the SPL_(N) values to a logarithmic curve and factoris a constant (e.g., 40) used to scale volume prior to computing the logfunction log f( ) Other values of factor could also be used (e.g., 10).

FIG. 10E illustrates a fifth GUI of a companion device 1000 for locatinga wireless accessory device, including a sound GUI with the beamformerof the companion device oriented in the opposite direction of thewireless accessory device, according to an embodiment. The soundmeasurement GUI includes an SPL_(N) value 1011 a that is numericallydisplayed on the screen (e.g., as text) and a bar 1012 a (e.g., adigital or analog sound pressure meter) showing graphically the SPL_(N)value.

FIG. 10F illustrates a sixth GUI of a companion device for locating awireless accessory device, including a sound GUI with the beamformer ofthe companion device oriented in the direction of the wireless accessorydevice, according to an embodiment. The sound measurement GUI includesan SPL_(N) value 1011 b that is numerically displayed on the screen(e.g., as text) and a bar 1012 b (e.g., a digital or analog soundpressure meter) showing graphically the SPL_(N) value.

Example Case for Storing Wireless Accessory Device

In an embodiment, the wireless accessory device may be stored in a case.If the case is lost, then a sound played through a loudspeaker of thewireless accessory device may be inaudible when stored in the case. Toaddress this problem, the case itself can include a loudspeaker and/orvibrator, a microprocessor, memory and a wireless transceiver. Thewireless transceiver can be a Bluetooth (BT) wireless transceiver chipor a hybrid BT/WiFi wireless transceiver chip that can handle both BTand WiFi protocols. The case can also include a charge connector forcharging the wireless accessory device and a power source (e.g., arechargeable battery) to power the electronics and wireless accessorydevice. The case can include one or more holes to allow sound from theloudspeaker out of the case. In an embodiment, the one or more holes canbe water-proofed using for example, rubber gaskets.

In an embodiment, the microprocessor can detect when the case is openedor closed and whether the wireless accessory device is stored in thecase. For example, the case can include a hinged-lid portion that isconfigured to provide a signal to the microprocessor when opened orclosed using, for example, metal contacts or proximity switches.Similarly, when the wireless accessory device is coupled to the chargingconnector, the charge connector can be configured to send a signal tothe microprocessor, which can be used to indicate that the wirelessaccessory device is charging in the case. The microprocessor can thenstore data (e.g., a binary code) in local memory that indicates theoperating status of the case. Timestamps indicating times of connectionand disconnection of the wireless accessory device to/from the chargeconnector can also be stored to assist in recovery of the wirelessaccessory device, as described below.

If the user loses their case, the user can use the processes previouslydescribed above to find their lost case. This includes, for example,playing a sound and using a sound locator GUI of a companion device tonavigate to the lost case, as described in reference to FIGS. 10A-10F.In an embodiment, the companion device (e.g., a smartphone, tablecomputer, wearable computer) can send a read request to the case over abi-directional wireless communication link (e.g., BT or WiFi) to readthe operating state of the case from local memory. The read request cancause the microprocessor to read from local memory one or more statuscodes and send the one or more status codes to the wireless transceiverchip for transmission to the companion device. In an embodiment, the oneor more status codes can indicate: 1) whether or not the wirelessaccessory device is currently charging in the case; 2) whether the caseis open or closed; and 3) the remaining battery life.

In an example use scenario, the one or more status codes can be used bythe companion device to determine whether to play a sound through thewireless accessory device loudspeaker or the case loudspeaker. Forexample, if a status code indicates that the wireless accessory deviceis charging in the case and the case is open, then a sound can be playedfrom the loudspeaker of the wireless accessory device. If, however, thecase is closed then a loudspeaker of the case can be used to play thesound. If the case has a vibrator, the companion device can command thevibrator to start vibrating, which may provide an audible sound that canbe heard by the user and used to locate the case. If the status codeindicates that the battery life in the case is low, then that statuscould be used as a factor for playing a sound through the loudspeaker ofthe wireless accessory device rather than the loudspeaker/vibrator ofthe case.

In an embodiment where the case includes a hybrid BT/WiFi chip, thelocation of the case may be estimated using WiFi positioning and storedin local memory. When the wireless accessory device is disconnected fromthe charging connector, the last known location of the wirelessaccessory device (as determined using WiFi positioning) is stored inlocal memory with a timestamp and displayed on a companion device uponrequest by the user. The user can then search for the lost wirelessaccessory device at the last known location of the case where the userpresumably removed the wireless accessory device from the case.

Example Server Architecture

FIG. 11 is a block diagram of example server architecture forimplementing the features and processes described in reference to FIGS.1-10, according to an embodiment. Other architectures are possible,including architectures with more or fewer components. In someimplementations, architecture 1100 includes one or more processor(s)1102 (e.g., dual-core Intel® Xeon® Processors), one or more networkinterface(s) 1106, one or more storage device(s) 1104 (e.g., hard disk,optical disk, flash memory) and one or more computer-readable medium(s)1108 (e.g., hard disk, optical disk, flash memory, etc.). Thesecomponents can exchange communications and data over one or morecommunication channel(s) 1110 (e.g., buses), which can utilize varioushardware and software for facilitating the transfer of data and controlsignals between components.

The term “computer-readable medium” refers to any medium thatparticipates in providing instructions to processor(s) 1102 forexecution, including without limitation, non-volatile media (e.g.,optical or magnetic disks), volatile media (e.g., memory) andtransmission media. Transmission media includes, without limitation,coaxial cables, copper wire and fiber optics.

Computer-readable medium(s) 1108 can further include operating system1112 (e.g., Mac OS® server, Windows® NT server), network communicationmodule 1114, remote location-tracking service 1116 and other services1118.

Operating system 1112 can be multi-user, multiprocessing, multitasking,multithreading, real time, etc. Operating system 1112 performs basictasks, including but not limited to: recognizing input from andproviding output to devices 1102, 1104, 1106 and 1108; keeping track andmanaging files and directories on computer-readable medium(s) 1108(e.g., memory or a storage device); controlling peripheral devices; andmanaging traffic on the one or more communication channel(s) 1110.Network communications module 1114 includes various components forestablishing and maintaining network connections (e.g., software forimplementing communication protocols, such as TCP/IP, HTTP, etc.).

In an embodiment, remote-tracking service 1116 implements a website thatallows a user of a requesting device to configure a lost mode on awireless device, as described in reference to FIG. 1. Service 1116 canalso perform a remote wipe of a companion device (erase its contents),track histories of companion and accessory devices and play soundsthrough companion and accessory devices. Other network services 1118 caninclude but are not limited to: map and navigation services,location-based services, device registration/authentication and useraccount administration.

Architecture 1100 can be included in any computer device, including oneor more server computers in a local or distributed network each havingone or more processing cores. Architecture 1100 can be implemented in aparallel processing or peer-to-peer infrastructure or on a single devicewith one or more processors. Software can include multiple softwarecomponents or can be a single body of code.

Example Companion Device Architecture

FIG. 12 is a block diagram of example companion device architecture 1200for implementing the features and processes described in reference toFIGS. 1-10. Architecture 1200 may be implemented in any electronicdevice for generating the features and processes described in referenceto FIGS. 1-3, including but not limited to smart phones, tabletcomputers and wearable computers (e.g., smart watches, fitness bands).Architecture 1200 may include memory interface 1202, data processor(s),image processor(s) or central processing unit(s) 1204, and peripheralsinterface 1206. Memory interface 1202, processor(s) 1204 or peripheralsinterface 1206 may be separate components or may be integrated in one ormore integrated circuits. One or more communication buses or signallines may couple the various components.

Sensors, devices, and subsystems may be coupled to peripherals interface1206 to facilitate multiple functionalities. For example, motionsensor(s) 1210, light sensor 1212, and proximity sensor 1214 may becoupled to peripherals interface 1206 to facilitate orientation,lighting, and proximity functions of the mobile device. For example, insome implementations, light sensor 1212 may be utilized to facilitateadjusting the brightness of touch surface 1246. In some implementations,motion sensor(s) 1210 (e.g., an accelerometer, rate gyroscope) may beutilized to detect movement and orientation of the device. Accordingly,display objects or media may be presented according to a detectedorientation (e.g., portrait or landscape).

Other sensors may also be connected to peripherals interface 1206, suchas a temperature sensor, a barometer, a biometric sensor or othersensing device, to facilitate related functionalities. For example, abiometric sensor can detect fingerprints and monitor heart rate andother fitness parameters. In an embodiment, a haptic motor (not shown)can be coupled to the peripheral interface, which can provide vibrationpatterns as haptic feedback.

Location processor 1215 (e.g., GNSS receiver chip) may be connected toperipherals interface 1206 to provide geo-referencing. Electronicmagnetometer 1216 (e.g., an integrated circuit chip) may also beconnected to peripherals interface 1206 to provide data that may be usedto determine the direction of magnetic North. Thus, electronicmagnetometer 1216 may be used as an electronic compass.

Camera subsystem 1220 and an optical sensor 1222, e.g., a chargedcoupled device (CCD) or a complementary metal-oxide semiconductor (CMOS)optical sensor, may be utilized to facilitate camera functions, such asrecording photographs and video clips.

Communication functions may be facilitated through one or morecommunication subsystems 1224. Communication subsystem(s) 1224 mayinclude one or more wireless communication subsystems. Wirelesscommunication subsystems 1224 may include radio frequency receivers andtransmitters and/or optical (e.g., infrared) receivers and transmitters.Wired communication systems may include a port device, e.g., a UniversalSerial Bus (USB) port or some other wired port connection that may beused to establish a wired connection to other computing devices, such asother communication devices, network access devices, a personalcomputer, a printer, a display screen, or other processing devicescapable of receiving or transmitting data.

The specific design and implementation of the communication subsystem1224 may depend on the communication network(s) or medium(s) over whichthe device is intended to operate. For example, a device may includewireless communication subsystems designed to operate over a globalsystem for mobile communications (GSM) network, a GPRS network, anenhanced data GSM environment (EDGE) network, IEEE802.xx communicationnetworks (e.g., Wi-Fi, Wi-Max, ZigBee™), 3G, 4G, 4G LTE, code divisionmultiple access (CDMA) networks, near field communication (NFC), Wi-FiDirect and a Bluetooth™ network. Wireless communication subsystems 1224may include hosting protocols such that the device may be configured asa base station for other wireless devices. As another example, thecommunication subsystems may allow the device to synchronize with a hostdevice using one or more protocols or communication technologies, suchas, for example, TCP/IP protocol, HTTP protocol, UDP protocol, ICMPprotocol, POP protocol, FTP protocol, IMAP protocol, DCOM protocol, DDEprotocol, SOAP protocol, HTTP Live Streaming, MPEG Dash and any otherknown communication protocol or technology.

Audio subsystem 1226 may be coupled to a speaker 1228 and one or moremicrophones 1230 to facilitate voice-enabled functions, such as voicerecognition, voice replication, digital recording, telephony functionsand for receiving sound signals from an accessory device, as describedin reference to FIGS. 1-10.

I/O subsystem 1240 may include touch controller 1242 and/or anotherinput controller(s) 1244. Touch controller 1242 may be coupled to atouch surface 1246. Touch surface 1246 and touch controller 1242 may,for example, detect contact and movement or break thereof using any of anumber of touch sensitivity technologies, including but not limited to,capacitive, resistive, infrared, and surface acoustic wave technologies,as well as other proximity sensor arrays or other elements fordetermining one or more points of contact with touch surface 1246. Inone implementation, touch surface 1246 may display virtual or softbuttons and a virtual keyboard, which may be used as an input/outputdevice by the user.

Other input controller(s) 1244 may be coupled to other input/controldevices 1248, such as one or more buttons, rocker switches, thumb-wheel,infrared port, USB port, and/or a pointer device such as a stylus. Theone or more buttons (not shown) may include an up/down button for volumecontrol of speaker 1228 and/or microphone 1230.

In some implementations, device 1200 may present recorded audio and/orvideo files, such as MP3, AAC, and MPEG video files. In someimplementations, device 1200 may include the functionality of an MP3player and may include a pin connector for tethering to other devices.Other input/output and control devices may be used. In an embodiment,device 1200 may include an audio processing unit for streaming audio toan accessory device over a direct or indirect communication link, asdescribed in reference to FIGS. 1-10.

Memory interface 1202 may be coupled to memory 1250. Memory 1250 mayinclude high-speed random access memory or non-volatile memory, such asone or more magnetic disk storage devices, one or more optical storagedevices, or flash memory (e.g., NAND, NOR). Memory 1250 may storeoperating system 1252, such as Darwin, RTXC, LINUX, UNIX, OS X, iOS,WINDOWS, or an embedded operating system such as VxWorks. Operatingsystem 1252 may include instructions for handling basic system servicesand for performing hardware dependent tasks. In some implementations,operating system 1252 may include a kernel (e.g., UNIX kernel).

Memory 1250 may also store communication instructions 1254 to facilitatecommunicating with one or more additional devices, one or more computersor servers, including peer-to-peer communications with wirelessaccessory devices, as described in reference to FIGS. 1-10.Communication instructions 1254 may also be used to select anoperational mode or communication medium for use by the device, based ona geographic location (obtained by the GPS/Navigation instructions 1268)of the device.

Memory 1250 may include graphical user interface instructions 1256 tofacilitate graphic user interface processing, including a touch modelfor interpreting touch inputs and gestures; sensor processinginstructions 1258 to facilitate sensor-related processing and functions;phone instructions 1260 to facilitate phone-related processes andfunctions; electronic messaging instructions 1262 to facilitateelectronic-messaging related processes and functions; web browsinginstructions 1264 to facilitate web browsing-related processes andfunctions; media processing instructions 1266 to facilitate mediaprocessing-related processes and functions; GPS/Navigation instructions1268 to facilitate GPS and navigation-related processes; camerainstructions 1270 to facilitate camera-related processes and functions;and accessory location instructions 1272 for locating accessory devices,as described in reference to FIGS. 1-10. The GPS/Navigation instructions1268 include instructions for estimating location, including but notlimited to an extended Kalman filter and other processes for estimatinglocation.

Each of the above identified instructions and applications maycorrespond to a set of instructions for performing one or more functionsdescribed above. These instructions need not be implemented as separatesoftware programs, procedures, or modules. Memory 1250 may includeadditional instructions or fewer instructions. Furthermore, variousfunctions of the device may be implemented in hardware and/or insoftware, including in one or more signal processing and/or applicationspecific integrated circuits (ASICs).

Example Accessory Device Architecture

FIG. 13 is a wireless accessory device architecture, according to anembodiment. Architecture 1300 includes memory interface 1301, processor1302, peripherals interface 1303, memory 1304, one or more inertialsensors 1305, IED 1306, wireless communication subsystem 1307, audiosubsystem 1308, one or more speakers 1309, one or more microphones 1310and input/output (I/O) subsystem 1311. Memory further includesinstructions 1312 and sound file(s) 1313. Memory interface 1301,processor 1302 and/or peripherals interface 1303 may be separatecomponents or may be integrated in one or more integrated circuits.Other accessory devices, such as smartwatches may include an on-wristdetector (OWD) (that uses an optical or proximity sensor, for example)instead of an IED. One or more communication buses or signal lines maycouple the various components.

Sensors, devices, and subsystems may be coupled to peripherals interface1303 to facilitate multiple functionalities. For example, inertialsensor(s) 1305 and IED 1306 may be coupled to peripherals interface 1303to facilitate orientation and proximity functions of the accessorydevice. In some implementations, inertial sensors 1305 (e.g., anaccelerometer, rate gyroscope) may be utilized to detect movement andorientation of the accessory device. For example, if the accessorydevice is an ear bud, headphone or headset, a multi-axis accelerometercan be used to identify head gestures. Also, the accelerometer can beconfigured to detect vibrations due to user input (e.g., tapping on theaccessory device) to activate or deactivate the accessory device orperform another function. Other sensors may also be connected toperipheral interfaced 1303, such as a magnetometer, temperature sensor,a barometer or a biometric sensor to facilitate related functionalities.For example, a biometric sensor can be used for authentication and/or tomonitor heart rate and/or other fitness parameters IED 1307 can includean optical sensor, proximity switch, biometric sensor or any othersuitable sensor for detecting when the accessory device is placed in orproximate to a user's ear.

Communication functions may be facilitated through wirelesscommunication subsystem 1307. Wireless communication subsystem 1307 mayinclude a RF transceiver for short range, peer-to-peer communicationwith a companion device as described in reference to FIGS. 1-10. Thespecific design and implementation of the wireless communicationsubsystem 1307 may depend on the communication network(s) or medium(s)over which the device is intended to operate. For example, the wirelesscommunication subsystem 1307 may facilitate communication over a globalsystem for mobile communications (GSM) network, a GPRS network, anenhanced data GSM environment (EDGE) network, IEEE802.xx communicationnetworks (e.g., Wi-Fi, Wi-Max, ZigBee™), 3G, 4G, 4G LTE, code divisionmultiple access (CDMA) networks, near field communication (NFC), Wi-FiDirect and a Bluetooth™ network. Wireless communication subsystem 1307may include a software stack for implementing wireless communicationprotocols.

Audio subsystem 1308 may be coupled to one or more loudspeakers 1309 andone or more microphones 1311 to facilitate voice-enabled functions, suchas voice recognition, voice replication, digital recording, andtelephony functions, and to perform processes an enable features asdescribed in reference to FIGS. 1-10.

I/O subsystem 1311 may include a touch controller and/or another inputcontroller. The touch controller may be coupled to a touch surface. Thetouch surface and touch controller may, for example, detect contact andmovement or break thereof using any of a number of touch sensitivitytechnologies, including but not limited to, capacitive, resistive,infrared, and surface acoustic wave technologies, as well as otherproximity sensor arrays or other elements for determining one or morepoints of contact with touch surface. Other input controllers may becoupled to other input/control devices, such as one or more buttons,rocker switches or thumb-wheel. The one or more buttons may include anup/down button for volume control of speaker(s) 1309 and/or microphones1311. In some implementations, the accessory device may play recordedaudio files (e.g., sound files or audio messages), such as MP3, AAC, andMPEG video files. In some implementations, the accessory device mayinclude the functionality of an audio player (e.g., MP3 player).

Memory interface 1301 may be coupled to memory 1304. Memory 1304 mayinclude high-speed random access memory or non-volatile memory, such asone or more magnetic disk storage devices, one or more optical storagedevices, or flash memory (e.g., NAND, NOR). Memory 1304 may storeinstructions 1312 and sound file(s) 1313. Instructions can includeoperating system instructions for handling basic system services and forperforming hardware dependent tasks. Instructions 1312 can facilitatecommunicating with one or more additional devices, one or more computersor servers, including peer-to-peer communications with a companiondevice as described in reference to FIGS. 1-10. Instructions 1312 canfacilitate GUI processing, inertial sensor processing and IEDprocessing. Instructions 1312 can respond to signals or commands from acompanion device to play sound file 1313 with increasing levels ofloudness to a maximum loudness but not exceeding a maximum loudnessallowed by government regulations. Memory 1304 may include additionalinstructions or fewer instructions. The various functions of theaccessory device may be implemented in hardware and/or in software,including in one or more signal processing and/or application specificintegrated circuits (ASICs).

The features described may be implemented in digital electroniccircuitry or in computer hardware, firmware, software, or incombinations of them. The features may be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device, for execution by a programmableprocessor; and method steps may be performed by a programmable processorexecuting a program of instructions to perform functions of thedescribed implementations by operating on input data and generatingoutput.

The described features may be implemented advantageously in one or morecomputer programs that are executable on a programmable system includingat least one programmable processor coupled to receive data andinstructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that may be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program may be written in anyform of programming language (e.g., Objective-C, Java), includingcompiled or interpreted languages, and it may be deployed in any form,including as a stand-alone program or as a module, component,subroutine, or other unit suitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors orcores, of any kind of computer. Generally, a processor will receiveinstructions and data from a read-only memory or a random-access memoryor both. The essential elements of a computer are a processor forexecuting instructions and one or more memories for storing instructionsand data. Generally, a computer may communicate with mass storagedevices for storing data files. These mass storage devices may includemagnetic disks, such as internal hard disks and removable disks;magneto-optical disks; and optical disks. Storage devices suitable fortangibly embodying computer program instructions and data include allforms of non-volatile memory, including by way of example, semiconductormemory devices, such as EPROM, EEPROM, and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor andthe memory may be supplemented by, or incorporated in, ASICs(application-specific integrated circuits). To provide for interactionwith a user the features may be implemented on a computer having adisplay device such as a CRT (cathode ray tube), LED (light emittingdiode) or LCD (liquid crystal display) display or monitor for displayinginformation to the author, a keyboard and a pointing device, such as amouse or a trackball by which the author may provide input to thecomputer.

One or more features or steps of the disclosed embodiments may beimplemented using an Application Programming Interface (API). An API maydefine on or more parameters that are passed between a callingapplication and other software code (e.g., an operating system, libraryroutine, function) that provides a service, that provides data, or thatperforms an operation or a computation. The API may be implemented asone or more calls in program code that send or receive one or moreparameters through a parameter list or other structure based on a callconvention defined in an API specification document. A parameter may bea constant, a key, a data structure, an object, an object class, avariable, a data type, a pointer, an array, a list, or another call. APIcalls and parameters may be implemented in any programming language. Theprogramming language may define the vocabulary and calling conventionthat a programmer will employ to access functions supporting the API. Insome implementations, an API call may report to an application thecapabilities of a device running the application, such as inputcapability, output capability, processing capability, power capability,communications capability, etc.

As described above, some aspects of the subject matter of thisspecification include gathering and use of data available from varioussources to improve services a mobile device can provide to a user. Thepresent disclosure contemplates that in some instances, this gathereddata may identify a particular location or an address based on deviceusage. Such personal information data can include location-based data,addresses, subscriber account identifiers, or other identifyinginformation.

The present disclosure further contemplates that the entitiesresponsible for the collection, analysis, disclosure, transfer, storage,or other use of such personal information data will comply withwell-established privacy policies and/or privacy practices. Inparticular, such entities should implement and consistently use privacypolicies and practices that are generally recognized as meeting orexceeding industry or governmental requirements for maintaining personalinformation data private and secure. For example, personal informationfrom users should be collected for legitimate and reasonable uses of theentity and not shared or sold outside of those legitimate uses. Further,such collection should occur only after receiving the informed consentof the users. Additionally, such entities would take any needed stepsfor safeguarding and securing access to such personal information dataand ensuring that others with access to the personal information dataadhere to their privacy policies and procedures. Further, such entitiescan subject themselves to evaluation by third parties to certify theiradherence to widely accepted privacy policies and practices.

In the case of advertisement delivery services, the present disclosurealso contemplates embodiments in which users selectively block the useof, or access to, personal information data. That is, the presentdisclosure contemplates that hardware and/or software elements can beprovided to prevent or block access to such personal information data.For example, in the case of advertisement delivery services, the presenttechnology can be configured to allow users to select to “opt in” or“opt out” of participation in the collection of personal informationdata during registration for services.

Therefore, although the present disclosure broadly covers use ofpersonal information data to implement one or more various disclosedembodiments, the present disclosure also contemplates that the variousembodiments can also be implemented without the need for accessing suchpersonal information data. That is, the various embodiments of thepresent technology are not rendered inoperable due to the lack of all ora portion of such personal information data. For example, content can beselected and delivered to users by inferring preferences based onnon-personal information data or a bare minimum amount of personalinformation, such as the content being requested by the deviceassociated with a user, other non-personal information available to thecontent delivery services, or publically available information.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. Elements of one ormore implementations may be combined, deleted, modified, or supplementedto form further implementations. In yet another example, the logic flowsdepicted in the figures do not require the particular order shown, orsequential order, to achieve desirable results. In addition, other stepsmay be provided, or steps may be eliminated, from the described flows,and other components may be added to, or removed from, the describedsystems. Accordingly, other implementations are within the scope of thefollowing claims.

What is claimed is: 1-20. (canceled)
 21. An electronic devicecomprising: a communications radio; one or more inertial sensors; adisplay; a wireless communications interface; one or more processors;and memory operable to store instructions, which, when executed by theone or more processors, causes the one or more processors to: determinean orientation of an electronic device based on sensor data provided bythe one or more inertial sensors of the electronic device; receive,using the wireless communications interface, a signal from an accessorydevice and determine at least one signal strength value from thereceived signal; estimate a proximity value to the accessory devicebased on the at least one signal strength value corresponding to theorientation; associate the orientation of the electronic device with theproximity value; and display, via the display, a graphical interface toindicate the proximity value to the accessory device for thecorresponding orientation of the electronic device.
 22. The electronicdevice of claim 21, the instructions further cause the one or moreprocessors to: send, to a server, the proximity value; and receive, fromthe server, a request to display on the graphical interface an option torequest sound play on the accessory device.
 23. The electronic device ofclaim 21, wherein the proximity value is indicated with a range classvalue.
 24. The electronic device of claim 23, wherein the range classvalue is calculated from at least one of a formula or a statisticalfunction.
 25. The electronic device of claim 21, the instructionsfurther cause the one or more processors to: receive, by two or moremicrophones of the electronic device, a recording of the environment,the recording including the sound played through the loudspeaker of theaccessory device and ambient noise; filter, by one or more filters ofthe electronic device, the recording, the one or more filters configuredto pass the specified frequency or frequencies of the sound playedthrough the loudspeaker of the accessory device and to reduce masking ofthe sound by the ambient noise; and determine, by the electronic device,a direction to the accessory device based on a determination of adirection of a sound source within the filtered recording of theenvironment, the sound source having the specified frequency orfrequencies; and display, via the display, a graphical interface toindicate the direction to the accessory device for the correspondingorientation of the electronic device.
 26. The electronic device of claim21, the instructions further cause the one or more processors to:determine the at least one received signal strength value using thewireless communications interface of the companion device.
 27. Theelectronic device of claim 21, the instructions further cause the one ormore processors to: determine a compass heading from the received sensordata.
 28. A method performed by an electronic device, comprising:determining an orientation of an electronic device based on sensor dataprovided by one or more inertial sensors of the electronic device;receiving, using a wireless communications interface, a signal from anaccessory device and determine at least one signal strength value fromthe received signal; estimating a proximity value to the accessorydevice based on the at least one signal strength value corresponding tothe orientation; associating the orientation of the electronic devicewith the proximity value; and displaying, via a display, a graphicalinterface to indicate the proximity value to the accessory device forthe corresponding orientation of the electronic device.
 29. The methodof claim 28, further comprising: sending, to a server, the proximityvalue; and receiving, from the server, a request to display on thegraphical interface an option to request sound play on the accessorydevice.
 30. The method of claim 28, wherein the proximity value isindicated with a range class value.
 31. The method of claim 28, whereinthe range class value is calculated from at least one of a formula or astatistical function.
 32. The method of claim 28, further comprising:receiving, by two or more microphones of the electronic device, arecording of the environment, the recording including the sound playedthrough the loudspeaker of the accessory device and ambient noise;filtering, by one or more filters of the electronic device, therecording, the one or more filters configured to pass the specifiedfrequency or frequencies of the sound played through the loudspeaker ofthe accessory device and to reduce masking of the sound by the ambientnoise; and determining, by the electronic device, a direction to theaccessory device based on a determination of a direction of a soundsource within the filtered recording of the environment, the soundsource having the specified frequency or frequencies; and displaying,via the display, a graphical interface to indicate the direction to theaccessory device for the corresponding orientation of the electronicdevice.
 33. The method of claim 28, further comprising: determining theat least one received signal strength value using the wirelesscommunications interface of the companion device.
 34. The method ofclaim 28, further comprising: determining a compass heading from thereceived sensor data.
 35. A non-transitory, computer-readable storagemedium having instructions stored thereon, which, when executed by oneor more processors of an electronic device, causes the one or moreprocessors to perform operations comprising: determine an orientation ofan electronic device based on sensor data provided by one or moreinertial sensors of the electronic device; receive, using a wirelesscommunications interface, a signal from an accessory device anddetermine at least one signal strength value from the received signal;estimate a proximity value to the accessory device based on the at leastone signal strength value corresponding to the orientation; associatethe orientation of the electronic device with the proximity value; anddisplay, via a display, a graphical interface to indicate the proximityvalue to the accessory device for the corresponding orientation of theelectronic device.
 36. The non-transitory, computer readable storagemedium of claim 35, the instructions further cause the one or moreprocessors to: send, to a server, the proximity value; and receive, fromthe server, a request to display on the graphical interface an option torequest sound play on the accessory device.
 37. The non-transitory,computer readable storage medium of claim 35, wherein the proximityvalue is indicated with a range class value.
 38. The non-transitory,computer readable storage medium of claim 37, wherein the range classvalue is calculated from at least one of a formula or a statisticalfunction.
 39. The non-transitory, computer readable storage medium ofclaim 35, the instructions further cause the one or more processors to:receive, by two or more microphones of the electronic device, arecording of the environment, the recording including the sound playedthrough the loudspeaker of the accessory device and ambient noise;filter, by one or more filters of the electronic device, the recording,the one or more filters configured to pass the specified frequency orfrequencies of the sound played through the loudspeaker of the accessorydevice and to reduce masking of the sound by the ambient noise; anddetermine, by the electronic device, a direction to the accessory devicebased on a determination of a direction of a sound source within thefiltered recording of the environment, the sound source having thespecified frequency or frequencies; and display, via the display, agraphical interface to indicate the direction to the accessory devicefor the corresponding orientation of the electronic device.
 40. Thenon-transitory, computer readable storage medium of claim 35, theinstructions further cause the one or more processors to: determine theat least one received signal strength value using the wirelesscommunications interface of the companion device.
 41. Thenon-transitory, computer readable storage medium of claim 35, theinstructions further cause the one or more processors to: determine acompass heading from the received sensor data.